High-dimensional Outlier Detection: the Subspace Method

" In view of all that we have said in the foregoing sections, the many obstacles we appear to have surmounted, what casts the pall over our victory celebration? It is the curse of dimensionality, a malediction that has plagued the scientist from the earliest days. " – Richard Bellman 1. Introduction Many real data sets are very high dimensional. In some scenarios, real data sets may contain hundreds or thousands of dimensions. With increasing dimensionality, many of the conventional outlier detection methods do not work very effectively. This is an artifact of the well known curse of dimensionality. In high-dimensional space, the data becomes sparse, and the true outliers become masked by the noise effects of multiple dimensions, when analyzed in full dimensionality. A main cause of the dimensionality curse is the difficulty in defining locality for the high dimensional case. For example, proximity-based methods define locality with the use of distance functions. On the other hand, it has been shown in [65, 215], that all pairs of points are almost equidistant in high-dimensional space. This is referred to as data spar­ sity. Since outliers are defined as data points in sparse regions, this results in a poorly discriminative situation where all data points are sit­ uated in an almost equally sparse regions in full dimensionality. The challenges arising from the dimensionality curse are not specific to out­ lier detection. It is well known that many problems such as clustering and similarity search experience qualitative challenges with increasing